A Comparison of Head Movement Classification Methods.

To understand human behavior, it is essential to study it in the context of natural movement in immersive, three-dimensional environments. Virtual reality (VR), with head-mounted displays, offers an unprecedented compromise between ecological validity and experimental control. However, such technological advancements mean that new data streams will become more widely available, and therefore, a need arises to standardize methodologies by which these streams are analyzed. One such data stream is that of head position and rotation tracking, now made easily available from head-mounted systems. The current study presents five candidate algorithms of varying complexity for classifying head movements. Each algorithm is compared against human rater classifications and graded based on the overall agreement as well as biases in metrics such as movement onset/offset time and movement amplitude. Finally, we conclude this article by offering recommendations for the best practices and considerations for VR researchers looking to incorporate head movement analysis in their future studies.

During natural viewing, neural processing of visual targets continues throughout saccades.

Relatively little is known about visual processing during free-viewing visual search in realistic dynamic environments. Free-viewing is characterized by frequent saccades. During saccades, visual processing is thought to be suppressed, yet we know that the presaccadic visual content can modulate postsaccadic processing. To better understand these processes in a realistic setting, we study here saccades and neural responses elicited by the appearance of visual targets in a realistic virtual environment. While subjects were being driven through a 3D virtual town, they were asked to discriminate between targets that appear on the road. Using a system identification approach, we separated overlapping and correlated activity evoked by visual targets, saccades, and button presses. We found that the presence of a target enhances early occipital as well as late frontocentral saccade-related responses. The earlier potential, shortly after 125 ms post-saccade onset, was enhanced for targets that appeared in the peripheral vision as compared to the central vision, suggesting that fast peripheral processing initiated before saccade onset. The later potential, at 195 ms post-saccade onset, was strongly modulated by the visibility of the target. Together these results suggest that, during natural viewing, neural processing of the presaccadic visual stimulus continues throughout the saccade, apparently unencumbered by saccadic suppression.

Gaze Behavior During Navigation and Visual Search of an Open-World Virtual Environment.

Eye tracking has been an essential tool within the vision science community for many years. However, the majority of studies involving eye-tracking technology employ a relatively passive approach through the use of static imagery, prescribed motion, or video stimuli. This is in contrast to our everyday interaction with the natural world where we navigate our environment while actively seeking and using task-relevant visual information. For this reason, an increasing number of vision researchers are employing virtual environment platforms, which offer interactive, realistic visual environments while maintaining a substantial level of experimental control. Here, we recorded eye movement behavior while subjects freely navigated through a rich, open-world virtual environment. Within this environment, subjects completed a visual search task where they were asked to find and count occurrence of specific targets among numerous distractor items. We assigned each participant into one of four target conditions: Humvees, motorcycles, aircraft, or furniture. Our results show a statistically significant relationship between gaze behavior and target objects across Target Conditions with increased visual attention toward assigned targets. Specifically, we see an increase in the number of fixations and an increase in dwell time on target relative to distractor objects. In addition, we included a divided attention task to investigate how search changed with the addition of a secondary task. With increased cognitive load, subjects slowed their speed, decreased gaze on objects, and increased the number of objects scanned in the environment. Overall, our results confirm previous findings and support that complex virtual environments can be used for active visual search experimentation, maintaining a high level of precision in the quantification of gaze information and visual attention. This study contributes to our understanding of how individuals search for information in a naturalistic (open-world) virtual environment. Likewise, our paradigm provides an intriguing look into the heterogeneity of individual behaviors when completing an un-timed visual search task while actively navigating.

"Blue Sky Effect": Contextual Influences on Pupil Size During Naturalistic Visual Search.

Pupil size is influenced by cognitive and non-cognitive factors. One of the strongest modulators of pupil size is scene luminance, which complicates studies of cognitive pupillometry in environments with complex patterns of visual stimulation. To help understand how dynamic visual scene statistics influence pupil size during an active visual search task in a visually rich 3D virtual environment (VE), we analyzed the correlation between pupil size and intensity changes of image pixels in the red, green, and blue (RGB) channels within a large window (~14 degrees) surrounding the gaze position over time. Overall, blue and green channels had a stronger influence on pupil size than the red channel. The correlation maps were not consistent with the hypothesis of a foveal bias for luminance, instead revealing a significant contextual effect, whereby pixels above the gaze point in the green/blue channels had a disproportionate impact on pupil size. We hypothesized this differential sensitivity of pupil responsiveness to blue light from above as a "blue sky effect," and confirmed this finding with a follow-on experiment with a controlled laboratory task. Pupillary constrictions were significantly stronger when blue was presented above fixation (paired with luminance-matched gray on bottom) compared to below fixation. This effect was specific for the blue color channel and this stimulus orientation. These results highlight the differential sensitivity of pupillary responses to scene statistics in studies or applications that involve complex visual environments and suggest blue light as a predominant factor influencing pupil size.

A Case for Studying Naturalistic Eye and Head Movements in Virtual Environments.

Using head mounted displays (HMDs) in conjunction with virtual reality (VR), vision researchers are able to capture more naturalistic vision in an experimentally controlled setting. Namely, eye movements can be accurately tracked as they occur in concert with head movements as subjects navigate virtual environments. A benefit of this approach is that, unlike other mobile eye tracking (ET) set-ups in unconstrained settings, the experimenter has precise control over the location and timing of stimulus presentation, making it easier to compare findings between HMD studies and those that use monitor displays, which account for the bulk of previous work in eye movement research and vision sciences more generally. Here, a visual discrimination paradigm is presented as a proof of concept to demonstrate the applicability of collecting eye and head tracking data from an HMD in VR for vision research. The current work's contribution is 3-fold: firstly, results demonstrating both the strengths and the weaknesses of recording and classifying eye and head tracking data in VR, secondly, a highly flexible graphical user interface (GUI) used to generate the current experiment, is offered to lower the software development start-up cost of future researchers transitioning to a VR space, and finally, the dataset analyzed here of behavioral, eye and head tracking data synchronized with environmental variables from a task specifically designed to elicit a variety of eye and head movements could be an asset in testing future eye movement classification algorithms.

Trust and Distrust of Automated Parking in a Tesla Model X.

OBJECTIVE: The present study aims to evaluate driver intervention behaviors during a partially automated parking task. BACKGROUND: Cars with partially automated parking features are becoming widely available. Although recent research explores the use of automation features in partially automated cars, none have focused on partially automated parking. Recent incidents and research have demonstrated that drivers sometimes use partially automated features in unexpected, inefficient, and harmful ways. METHOD: Participants completed a series of partially automated parking trials with a Tesla Model X and their behavioral interventions were recorded. Participants also completed a risk-taking behavior test and a post-experiment questionnaire that included questions about trust in the system, likelihood of using the Autopark feature, and preference for either the partially automated parking feature or self-parking. RESULTS: Initial intervention rates were over 50%, but declined steeply in later trials. Responses to open-ended questions revealed that once participants understood what the system was doing, they were much more likely to trust it. Trust in the partially automated parking feature was predicted by a model including risk-taking behaviors, self-confidence, self-reported number of errors committed by the Tesla, and the proportion of trials in which the driver intervened. CONCLUSION: Using partially automated parking with little knowledge of its workings can lead to high degree of initial distrust. Repeated exposure of partially automated features to drivers can greatly increase their use. APPLICATION: Short tutorials and brief explanations of the workings of partially automated features may greatly improve trust in the system when drivers are first introduced to partially automated systems.

Decoding P300 Variability Using Convolutional Neural Networks.

Deep convolutional neural networks (CNN) have previously been shown to be useful tools for signal decoding and analysis in a variety of complex domains, such as image processing and speech recognition. By learning from large amounts of data, the representations encoded by these deep networks are often invariant to moderate changes in the underlying feature spaces. Recently, we proposed a CNN architecture that could be applied to electroencephalogram (EEG) decoding and analysis. In this article, we train our CNN model using data from prior experiments in order to later decode the P300 evoked response from an unseen, hold-out experiment. We analyze the CNN output as a function of the underlying variability in the P300 response and demonstrate that the CNN output is sensitive to the experiment-induced changes in the neural response. We then assess the utility of our approach as a means of improving the overall signal-to-noise ratio in the EEG record. Finally, we show an example of how CNN-based decoding can be applied to the analysis of complex data.

The fixation-related lambda response: Effects of saccade magnitude, spatial frequency, and ocular artifact removal.

Fixation-related potentials (FRPs) enable examination of electrophysiological signatures of visual perception under naturalistic conditions, providing a neural snapshot of the fixated scene. The most prominent FRP component, commonly referred to as the lambda response, is a large deflection over occipital electrodes (O1, Oz, O2) peaking 80-100 ms post fixation, reflecting afferent input to visual cortex. The lambda response is affected by bottom-up stimulus features and the size of the preceding saccade; however, prior research has not adequately controlled for these influences in free viewing paradigms. The current experiment (N = 16, 1 female) addresses these concerns by systematically manipulating spatial frequency in a free-viewing task requiring a range of saccade sizes. Given the close temporal proximity between saccade related activity and the onset of the lambda response, we evaluate how removing independent components (IC) associated with ocular motion artifacts affects lambda response amplitude. Our results indicate that removing ocular artifact ICs based on the covariance with gaze position did not significantly affect the amplitude of this occipital potential. Moreover, the results showed that spatial frequency and saccade magnitude each produce significant effects on lambda amplitude, where amplitude decreased with increasing spatial frequency and increased as a function of saccade size for small and medium-sized saccades. The amplitude differences between spatial frequencies were maintained across all saccade magnitudes suggesting these effects are produced from distinctly different and uncorrelated mechanisms. The current results will inform future analyses of the lambda potential in natural scenes where saccade magnitudes and spatial frequencies ultimately vary.

Isolating Discriminant Neural Activity in the Presence of Eye Movements and Concurrent Task Demands.

A growing number of studies use the combination of eye-tracking and electroencephalographic (EEG) measures to explore the neural processes that underlie visual perception. In these studies, fixation-related potentials (FRPs) are commonly used to quantify early and late stages of visual processing that follow the onset of each fixation. However, FRPs reflect a mixture of bottom-up (sensory-driven) and top-down (goal-directed) processes, in addition to eye movement artifacts and unrelated neural activity. At present there is little consensus on how to separate this evoked response into its constituent elements. In this study we sought to isolate the neural sources of target detection in the presence of eye movements and over a range of concurrent task demands. Here, participants were asked to identify visual targets (Ts) amongst a grid of distractor stimuli (Ls), while simultaneously performing an auditory N-back task. To identify the discriminant activity, we used independent components analysis (ICA) for the separation of EEG into neural and non-neural sources. We then further separated the neural sources, using a modified measure-projection approach, into six regions of interest (ROIs): occipital, fusiform, temporal, parietal, cingulate, and frontal cortices. Using activity from these ROIs, we identified target from non-target fixations in all participants at a level similar to other state-of-the-art classification techniques. Importantly, we isolated the time course and spectral features of this discriminant activity in each ROI. In addition, we were able to quantify the effect of cognitive load on both fixation-locked potential and classification performance across regions. Together, our results show the utility of a measure-projection approach for separating task-relevant neural activity into meaningful ROIs within more complex contexts that include eye movements.

EEG and Eye Tracking Signatures of Target Encoding during Structured Visual Search.

EEG and eye tracking variables are potential sources of information about the underlying processes of target detection and storage during visual search. Fixation duration, pupil size and event related potentials (ERPs) locked to the onset of fixation or saccade (saccade-related potentials, SRPs) have been reported to differ dependent on whether a target or a non-target is currently fixated. Here we focus on the question of whether these variables also differ between targets that are subsequently reported (hits) and targets that are not (misses). Observers were asked to scan 15 locations that were consecutively highlighted for 1 s in pseudo-random order. Highlighted locations displayed either a target or a non-target stimulus with two, three or four targets per trial. After scanning, participants indicated which locations had displayed a target. To induce memory encoding failures, participants concurrently performed an aurally presented math task (high load condition). In a low load condition, participants ignored the math task. As expected, more targets were missed in the high compared with the low load condition. For both conditions, eye tracking features distinguished better between hits and misses than between targets and non-targets (with larger pupil size and shorter fixations for missed compared with correctly encoded targets). In contrast, SRP features distinguished better between targets and non-targets than between hits and misses (with average SRPs showing larger P300 waveforms for targets than for non-targets). Single trial classification results were consistent with these averages. This work suggests complementary contributions of eye and EEG measures in potential applications to support search and detect tasks. SRPs may be useful to monitor what objects are relevant to an observer, and eye variables may indicate whether the observer should be reminded of them later.

Best practice for single-trial detection of event-related potentials: Application to brain-computer interfaces.

The detection of event-related potentials (ERPs) in the electroencephalogram (EEG) signal is a fundamental component in non-invasive brain-computer interface (BCI) research, and in modern cognitive neuroscience studies. Whereas the grand average response across trials provides an estimation of essential characteristics of a brain-evoked response, an estimation of the differences between trials for a particular type of stimulus can provide key insight about the brain dynamics and possible origins of the brain response. The research in ERP single-trial detection has been mainly driven by applications in biomedical engineering, with an interest from machine learning and signal processing groups that test novel methods on noisy signals. Efficient single-trial detection techniques require processing steps that include temporal filtering, spatial filtering, and classification. In this paper, we review the current state-of-the-art methods for single-trial detection of event-related potentials with applications in BCI. Efficient single-trial detection techniques should embed simple yet efficient functions requiring as few hyper-parameters as possible. The focus of this paper is on methods that do not include a large number of hyper-parameters and can be easily implemented with datasets containing a limited number of trials. A benchmark of different classification methods is proposed on a database recorded from sixteen healthy subjects during a rapid serial visual presentation task. The results support the conclusion that single-trial detection can be achieved with an area under the ROC curve superior to 0.9 with less than ten sensors and 20 trials corresponding to the presentation of a target. Whereas the number of sensors is not a key element for efficient single-trial detection, the number of trials must be carefully chosen for creating a robust classifier.

The Impact of Task Demands on Fixation-Related Brain Potentials during Guided Search.

Recording synchronous data from EEG and eye-tracking provides a unique methodological approach for measuring the sensory and cognitive processes of overt visual search. Using this approach we obtained fixation related potentials (FRPs) during a guided visual search task specifically focusing on the lambda and P3 components. An outstanding question is whether the lambda and P3 FRP components are influenced by concurrent task demands. We addressed this question by obtaining simultaneous eye-movement and electroencephalographic (EEG) measures during a guided visual search task while parametrically modulating working memory load using an auditory N-back task. Participants performed the guided search task alone, while ignoring binaurally presented digits, or while using the auditory information in a 0, 1, or 2-back task. The results showed increased reaction time and decreased accuracy in both the visual search and N-back tasks as a function of auditory load. Moreover, high auditory task demands increased the P3 but not the lambda latency while the amplitude of both lambda and P3 was reduced during high auditory task demands. The results show that both early and late stages of visual processing indexed by FRPs are significantly affected by concurrent task demands imposed by auditory working memory.

A Permutation Test for Unbalanced Paired Comparisons of Global Field Power.

Global field power is a valuable summary of multi-channel electroencephalography data. However, global field power is biased by the noise typical of electroencephalography experiments, so comparisons of global field power on data with unequal noise are invalid. Here, we demonstrate the relationship between the number of trials that contribute to a global field power measure and the expected value of that global field power measure. We also introduce a statistical testing procedure that can be used for multi-subject, repeated-measures (also called within-subjects) comparisons of global field power when the number of trials per condition is unequal across conditions. Simulations demonstrate the effect of unequal trial numbers on global field power comparisons and show the validity of the proposed test in contrast to conventional approaches. Finally, the proposed test and two alternative tests are applied to data collected in a rapid serial visual presentation target detection experiment. The results show that the proposed test finds global field power differences in the classical P3 range; the other tests find differences in that range but also at other times including at times before stimulus onset. These results are interpreted as showing that the proposed test is valid and sensitive to real within-subject differences in global field power in multi-subject unbalanced data.

Common EEG features for behavioral estimation in disparate, real-world tasks.

In this study we explored the potential for capturing the behavioral dynamics observed in real-world tasks from concurrent measures of EEG. In doing so, we sought to develop models of behavior that would enable the identification of common cross-participant and cross-task EEG features. To accomplish this we had participants perform both simulated driving and guard duty tasks while we recorded their EEG. For each participant we developed models to estimate their behavioral performance during both tasks. Sequential forward floating selection was used to identify the montage of independent components for each model. Linear regression was then used on the combined power spectra from these independent components to generate a continuous estimate of behavior. Our results show that oscillatory processes, evidenced in EEG, can be used to successfully capture slow fluctuations in behavior in complex, multi-faceted tasks. The average correlation coefficients between the actual and estimated behavior was 0.548 ± 0.117 and 0.701 ± 0.154 for the driving and guard duty tasks respectively. Interestingly, through a simple clustering approach we were able to identify a number of common components, both neural and eye-movement related, across participants and tasks. We used these component clusters to quantify the relative influence of common versus participant-specific features in the models of behavior. These findings illustrate the potential for estimating complex behavioral dynamics from concurrent measures from EEG using a finite library of universal features.

The effect of target and non-target similarity on neural classification performance: a boost from confidence.

Brain computer interaction (BCI) technologies have proven effective in utilizing single-trial classification algorithms to detect target images in rapid serial visualization presentation tasks. While many factors contribute to the accuracy of these algorithms, a critical aspect that is often overlooked concerns the feature similarity between target and non-target images. In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes. It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented. This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets. The resulting findings show that behavior is slower and less accurate when targets are presented together with similar non-targets; moreover, single-trial classification yielded high levels of misclassification when infrequent non-targets are included. Furthermore, we present an approach to mitigate the image misclassification. We use confidence measures to assess the quality of single-trial classification, and demonstrate that a system in which low confidence trials are reclassified through a secondary process can result in improved performance.

Optimization of Single-Trial Detection of Event-Related Potentials Through Artificial Trials.

GOAL: Many brain-computer interface (BCI) classification techniques rely on a large number of labeled brain responses to create efficient classifiers. A large database representing all of the possible variability in the signal is impossible to obtain in a short period of time, and prolonged calibration times prevent efficient BCI use. We propose to improve BCIs based on the detection of event-related potentials (ERPs) in two ways. METHODS: First, we increase the size of the training database by considering additional deformed trials. The creation of the additional deformed trials is based on the addition of Gaussian noise, and on the variability of the ERP latencies. Second, we exploit the variability of the ERP latencies by combining decisions across multiple deformed trials. These new methods are evaluated on data from 16 healthy subjects participating in a rapid serial visual presentation task. RESULTS: The results show a significant increase in the performance of single-trial detection with the addition of artificial trials, and the combination of decisions obtained from altered trials. When the number of trials to train a classifier is low, the proposed approach allows us improve performance from an AUC of 0.533±0.080 to 0.905±0.053. This improvement represents approximately an 80% reduction in classification error. CONCLUSION: These results demonstrate that artificially increasing the training dataset leads to improved single-trial detection. SIGNIFICANCE: Calibration sessions can be shortened for BCIs based on ERP detection.

Usability of four commercially-oriented EEG systems.

Electroencephalography (EEG) holds promise as a neuroimaging technology that can be used to understand how the human brain functions in real-world, operational settings while individuals move freely in perceptually-rich environments. In recent years, several EEG systems have been developed that aim to increase the usability of the neuroimaging technology in real-world settings. Here, the usability of three wireless EEG systems from different companies are compared to a conventional wired EEG system, BioSemi's ActiveTwo, which serves as an established laboratory-grade 'gold standard' baseline. The wireless systems compared include Advanced Brain Monitoring's B-Alert X10, Emotiv Systems' EPOC and the 2009 version of QUASAR's Dry Sensor Interface 10-20. The design of each wireless system is discussed in relation to its impact on the system's usability as a potential real-world neuroimaging system. Evaluations are based on having participants complete a series of cognitive tasks while wearing each of the EEG acquisition systems. This report focuses on the system design, usability factors and participant comfort issues that arise during the experimental sessions. In particular, the EEG systems are assessed on five design elements: adaptability of the system for differing head sizes, subject comfort and preference, variance in scalp locations for the recording electrodes, stability of the electrical connection between the scalp and electrode, and timing integration between the EEG system, the stimulus presentation computer and other external events.

Estimating endogenous changes in task performance from EEG.

Brain wave activity is known to correlate with decrements in behavioral performance as individuals enter states of fatigue, boredom, or low alertness.Many BCI technologies are adversely affected by these changes in user state, limiting their application and constraining their use to relatively short temporal epochs where behavioral performance is likely to be stable. Incorporating a passive BCI that detects when the user is performing poorly at a primary task, and adapts accordingly may prove to increase overall user performance. Here, we explore the potential for extending an established method to generate continuous estimates of behavioral performance from ongoing neural activity; evaluating the extended method by applying it to the original task domain, simulated driving; and generalizing the method by applying it to a BCI-relevant perceptual discrimination task. Specifically, we used EEG log power spectra and sequential forward floating selection (SFFS) to estimate endogenous changes in behavior in both a simulated driving task and a perceptual discrimination task. For the driving task the average correlation coefficient between the actual and estimated lane deviation was 0.37 ± 0.22 (µ ± σ). For the perceptual discrimination task we generated estimates of accuracy, reaction time, and button press duration for each participant. The correlation coefficients between the actual and estimated behavior were similar for these three metrics (accuracy = 0.25 ± 0.37, reaction time = 0.33 ± 0.23, button press duration = 0.36 ± 0.30). These findings illustrate the potential for modeling time-on-task decrements in performance from concurrent measures of neural activity.

Sliding HDCA: single-trial EEG classification to overcome and quantify temporal variability.

Patterns of neural data obtained from electroencephalography (EEG) can be classified by machine learning techniques to increase human-system performance. In controlled laboratory settings this classification approach works well; however, transitioning these approaches into more dynamic, unconstrained environments will present several significant challenges. One such challenge is an increase in temporal variability in measured behavioral and neural responses, which often results in suboptimal classification performance. Previously, we reported a novel classification method designed to account for temporal variability in the neural response in order to improve classification performance by using sliding windows in hierarchical discriminant component analysis (HDCA), and demonstrated a decrease in classification error by over 50% when compared to the standard HDCA method (Marathe et al., 2013). Here, we expand upon this approach and show that embedded within this new method is a novel signal transformation that, when applied to EEG signals, significantly improves the signal-to-noise ratio and thereby enables more accurate single-trial analysis. The results presented here have significant implications for both brain-computer interaction technologies and basic science research into neural processes.

Multiclass classification of single-trial evoked EEG responses.

The detection of event-related potentials (ERPs) in the electroencephalogram (EEG) signal has several real-world applications, from cognitive state monitoring to brain-computer interfaces. Current systems based on the detection of ERPs only consider a single type of response to detect. Hence, the classification methods that are considered for ERP detection are binary classifiers (target vs. non target). Here we investigated multiclass classification of single-trial evoked responses during a rapid serial visual presentation task in which short video clips were presented to fifteen observers. Each trial contained potential targets that were human or non-human, stationary or moving. The goal of the classification analysis was to discriminate between three classes: moving human targets, moving non-human targets, and non-moving human targets. The analysis revealed that the mean volume under the ROC surface of 0.878. These results suggest that it is possible to efficiently discriminate between more than two types of evoked responses using single-trial detection.